For the applications of face recognition, a learning based kernel image differential filter was proposed. Firstly, instead of designing the image filter in a handcrafted or analytical way, the new image filter was designed by dynamically learning from the training data. By integrating the idea of Linear Discriminant Analysis (LDA) into filter learning, the intra-class difference of filtered image was attenuated and the inter-class difference was amplified. Secondly, the second order derivative operator and kernel trick were introduced to better extract the image detail information and cope with the nonlinear feature space problem. As a result, the filter is adaptive and more discriminative feature description can be obtained. The proposed algorithm was experimented on AR and ORL face database and compared with linearly learning image filter named IFL, kernel image filter without differential information, and kernel image filter considering only one order differential information. The experimental results validate the effectiveness of the proposed method.